There are two different modes in video that rarely existed in images, but now that video and image share formats, they have become relevant. What's often called full-range, PC-mode, or JPEG-mode is from 0-255, while limited-range, TV-mode, or video-mode is from 16-235 (16-240 chroma), with those numbers scaled up for 10+ bit. The latter is by far the most common for any video you'll encounter, the former for images.

Libheif supports both via the full_range_flag in nclx, and will flag an image as being in limited or full range -- the decoder should handle that. (Which decoders have been getting much better about actually doing.) I think you can override it to get the right color conversion even when decoding. That saves the hit of an extra color conversion step plus additional round-off errors from the scaling.

I tried to use that information to update the approach, but wasn't able to get something fully working.

Here's what I tried:

Create the nclx color profile:

  struct heif_color_profile_nclx* nclx = heif_nclx_color_profile_alloc();
  nclx->color_primaries = get_heif_primaries(primaries);
  nclx->transfer_characteristics = get_heif_transfer(transfer);
  nclx->matrix_coefficients = get_heif_matrix(matrix);
  nclx->full_range_flag = is_full_range ? 1 : 0;
  heif_image_set_nclx_color_profile(*out_img, nclx);
  heif_nclx_color_profile_free(nclx);

Implemented conversion functions to convert from the values returned by the browser's decoded VideoFrame:

static heif_color_primaries get_heif_primaries(const std::string& p) {
  if (p == "bt709") return heif_color_primaries_ITU_R_BT_709_5;
  if (p == "smpte170m") return heif_color_primaries_ITU_R_BT_601_6;
  // output truncated for simplicity
}

static heif_transfer_characteristics get_heif_transfer(const std::string& t) {
  if (t == "bt709") return heif_transfer_characteristic_ITU_R_BT_709_5;
  if (t == "smpte170m") return heif_transfer_characteristic_ITU_R_BT_601_6;
  // output truncated for simplicity
}

static heif_matrix_coefficients get_heif_matrix(const std::string& m) {
  if (m == "bt709") return heif_matrix_coefficients_ITU_R_BT_709_5;
  if (m == "smpte170m") return heif_matrix_coefficients_ITU_R_BT_601_6;
  // output truncated for simplicity
}

It would also be possible to grab this data from the VUI section of the SPS data, but the SPS parsing function we've got in this file doesn't have code to parse the VUI info.

I printed some debug output to make sure this was all getting set properly, and I saw primaries = smpte170m (6), transfer = bt709 (1), matrix = smpte170m (6), and full range = 0.

All these values seem correct to me, so I'm not quite sure why it doesn't work. Is there anything special you have to do when using the image data, such as over in post.js?

I've investigated this further and discovered two things that prevent setting of the nclx profile from working as expected.

• The color conversion functions in yuv2rgb.cc don't re-map the alpha channel from limited to full range. This causes fully transparent alpha channels to be light gray even when a conversion function that handles limited range data is selected.

• I am seeing that even if I return nclx data on the decoded image, libheif may ignore it. I tried to dig into this and I think what's going on is that if a HEIF image contains nclx data in the colr box, it will be used instead of the nclx data from the decoded tiles. This can cause it to incorrectly think that the image is using full range, even if the decoded image data is using limited range. I am testing with an image generated from an iPhone that has an alpha channel and am seeing that Op_YCbCr420_to_RGB32 is being selected for color conversion even though the decoded tiles are in limited range and Op_YCbCr420_to_RGB32 only handles full range. In particular I am seeing that decoder_webcodecs.cc is setting primaries = 6, transfer = 1, matrix = 6, full range = 0 but yuv2rgb.cc is instead seeing primaries = 1, transfer = 13, matrix = 6, full range = 1.

After this investigation I can see that it doesn't make sense to do the limited range color conversion in the decoder plugin. So I will go ahead and update this PR just to properly set the nclx profile info on the decoded image.

However, I still think both things I raised above should be addressed to make color conversion & preserve the alpha channel properly for data returned from the webcodecs plugin. If there's any direction on that and whether you'd like me to take either thing on let me know.

Thanks!

@scriby Could you please attach the image with the limited-range alpha channel? Do you know which software created this?

The HEIF standard does not tell anything about the range of the alpha data. I can understand that images originally converted from JPEG use the limited range also in the HEIF file for the color channels, but I assumed that the alpha plane will always be full-range. I also see no way how a decoder could decide which range is correct. The colr color profile has the full-range flag, but my interpretation is that this only applies to the color channels. Since there is no alpha channel in legacy JPEG images or analog content, I see no reason why a HEIF file should store alpha with limited range.

2521.heic.zip

I created this file by taking a picture on an iphone, long pressing on a subject in the photo, copying it, and then pasting into iMessage and sending the message.

This image decodes properly with a transparent background when using libde265. So I'm not sure whether the original image is using limited range, or whether the browser decoding APIs are just producing limited range data anyway.

Update: I tested several HEIC images and they all decode with colorSpace.fullRange = false. I am pretty sure that the browser's HEVC decoding API is just always returning limited range colors no matter what the original input is. It also seems possible this is hardware dependent.

Thanks for the example image. This image sets full_range=true in the colr color profile for the color image:

| | | Box: colr -----
| | | size: 19   (header size: 8)
| | | colour_type: nclx
| | | colour_primaries: 2
| | | transfer_characteristics: 2
| | | matrix_coefficients: 6
| | | full_range_flag: 1

For the alpha plane, there is no colr box and hence no full_range information, as expected. The encoded data is full-range. Thus, I also think that the browser's decoder just clamps the output (wrongly) to 16-235.

The question is still where the decoder gets the full_range=false from.
It might also be that the browser decoder fills the colorspace information from the H.265 bitstream VUI and that this does not match the colr box in the HEIF container. The colr box takes precedence if both do not agree, but the browser decoder only sees the H.265 bitstream and might thus clamp the values. In that case, the encoder might be the guilty part, as it does not save a correct VUI header into the H.265 bitstream.

The best way to handle this seems to be to work around this browser bug by scaling it back to 0-255 in the decoder plugin.

BTW: I see that there is a alpha-channel metadata box alpp under consideration for HEIF 4th edition:
https://www.mpeg.org/wp-content/uploads/mpeg_meetings/151_Daejeon/w25291.zip
https://www.mpeg.org/wp-content/uploads/mpeg_meetings/151_Daejeon/w25442.zip

Instead of a full-range flag, there might be the values opaque_value / transparent_value, which seem to define the alpha range. However, this might be something for the future, but not for now.

PS: I had a look into the VUIs of the H.265 bitstreams of your example image. The VUIs are correct.
For color tiles: colour_primaries=2, transfer_characteristics=2, matrix_coefficients=6, full_range_flag=1
For alpha tiles: colour_primaries=2, transfer_characteristics=2, matrix_coefficients=2, full_range_flag=1

This matches the colr and indicates full-range for alpha. If the browser returns full_range=0, it is incorrect.

Heads up that this doesn't just affect the alpha channel. The actual color data is also using a limited range. IIRC the color data is using 16 - 240 and the alpha channel uses 16 - 235.

So even the normal colors of browser decoded images are off (at least using my hardware, I haven't been able to test this across different hardware yet).

I think there are 2 main ways to resolve it:

1. Use the nclx color profile information set by the decoder plugin on the returned heif image to map the colors in yuv2rgb.cc.

I've noticed that even though I'm setting nclx color profile info on the decoded image (similar to the decoder_libde265 plugin), it's not actually used by libheif. I think libheif may be defaulting to the info from the HEIF container's metadata.

This would also require an update to the yuv2rgb code to also map the alpha channel, which you might not want in general. It could be added as a setting on the nclx color profile data though (default off).

2. Map colors from limited to full range in the plugin.

As it is now, the plugin doesn't have enough information to know whether it should re-map or not. It would need to check for whether there is a mismatch between the full range flag that was returned by the decoder and what is specified on the colr box, or possibly the VUI info from SPS. The SPS parsing code in the plugin doesn't parse the VUI data, so it would need to be updated to read the full range flag out of VUI.

Investigating into the bug a little more, it looks like the hardware decoder on macOS may always returns limited range colors for HEVC streams: mpv-player/mpv#6546

Do you know whether the browser decoder scales the output values or whether it just clips them to the limited range?
I see no reason why it should scale it. I think it is more likely that it just clips the values. In that case, we might make things even worse by scaling the values.

Just by looking at it I think it looks scaled as opposed to clamped.

I'll do some comparisons with the raw output, the scaled output, and what libde265 returns to make sure.

I've got some interesting results from testing.

The overall result is that this PR improves colors for some images with the webcodec decoder when the decoder returns limited range colors (such as on my mac and perhaps every mac).

Comparing the colr boxes between the image that works well (go board.heic.zip) and the one that doesn't work well (2521.heic), the difference is that 2521.heic has colour_type = nclx and "go board.heic" has
colour_type = prof.

Best guess is that when the overall colour_type is prof, the nclx color profile from the decoder will be used, whereas when there is already a nclx profile in the container, it will be used instead.

As far as solving the issue, my sense is that libheif should use the nclx profile that the decoder sets (if present), and otherwise falls back to the one in the heif metadata.

That would at least allow libheif to re-map the color data in this case. As for fixing up the alpha channel, the plugin itself could do that if libheif doesn't think it's the right call to scale the alpha channel in general

Adding if (!img->has_nclx_color_profile()) { around that code resolves the color issues with 2521.heic by using the color profile returned by the decoder. It doesn't resolve the alpha channel issues, but that was still expected.

I can see in the comment that it was an intentional choice to to prefer the nclx profile provided in the HEIF metadata if both are present. I'm not sure if something else would be impacted by making this change.

Given this info do you have any guidance for how you'd like to proceed?

I can see in the comment that it was an intentional choice to to prefer the nclx profile provided in the HEIF metadata if both are present. I'm not sure if something else would be impacted by making this change.

FWIW, this is required by ISO/IEC 14496-12:2022 Section 12.1.5.1:

If colour information is supplied in both this box, and also in the video bitstream, this box takes precedence, and over-rides the information in the bitstream.

ISO/IEC 23008-12 Section 6.5.5 just points back to 14496-12 for colr syntax and semantics, although there are some variations in multiple colr boxes are handled.

Thanks for that info @bradh.

I'll try to explain why I think it doesn't apply (at least, directly) in this case.

In the 2521.heic file, the colr box and SPS VUI data have matching nclx color profiles. So there's no mismatch in terms of input data and either could be used.

The issue is that the decoder (in this case, device hardware) is for whatever reason changing the color profile when decoding.

So now the "ground truth" about which color profile is being used has changed, and if we use the color profile in the HEIF metadata it will no longer be correct.

Apparently this is the first decoder plugin which exhibits this sort of behavior, and the expectation up until now was that decoders would return color information in the same profile as the input source.

I don't know whether it's a spec violation or whatnot for a decoder to change the color profile when decoding, but there's not much I can do about that and just need to figure out what to do with the output.

I think we agree that the browser decoder illegally changes the decoded YCbCr values. It would not matter much if it just returned wrong VUI/nclx parameters, but as you said, it actually changes the decoded pixel values.
Fixing this by introducing a matching error in libheif core by using the decoder's nclx instead of the HEIF colr seems like a bad idea. If the decoder is known to modify the output, this has to be compensated by the plugin.

Our real problem is that we do not understand yet what the decoder is actually doing and why it is doing that.
We might have to check the behavior across several browsers and different graphics cards.
Or we could file an issue in the browser repo to clarify what is happening there.

Unfortunately, I didn't find the time yet to run this on my computers. But if you had a version online (e.g. https://strukturag.github.io/libheif/ with the new plugin), it should be easy to collect information on different browser/graphics card behavior.

I've pushed an update to the pull request to convert limited range to full range in the plugin if the result from the webcodecs API indicates it's using limited range colors.

It works well in my local testing but I'll see if I can get something hosted so you can see it as well.

Alright, I've got a hosted version of this PR up at https://scriby.github.io/libheif/.

I just made one modification to print out a debugging line to make it easier to see what the browser API is returning. Check for something like this below the image:

libheif webcodecs decoded. format: NV12, full range: false, primaries: bt709, matrix: bt709, transfer: bt709

I tried it out on a few different devices.

Chrome on Mac, Android, and Windows all returned NV12 data with full range: false.

The only place I've seen full range: true so far is in Safari on both Mac and iOS.

Update: I did some more testing on Windows. I found a HEIC image online for which Chrome will decode with full range: true, but my iPhone photos decode as full range: false.

In Edge, both the image I found online and my iPhone photos decode as full range: true. However, the sample images decode as full range: false and also have some glitches / corruption.

Neither Chrome nor Edge on my Windows machine will decode 2521.heic at all (image with alpha channel).